Conventionally, an imaging device such as a digital camera which captures an image by means of visible light is provided with an infrared cut filter to cut infrared light, and captures an image by means of only visible light. Meanwhile, an imaging device which is used for monitoring cameras and the like, and has an active sensor which captures images by radiating infrared light captures images by means of visible light and infrared light without an infrared cut filter. These images captured by means of visible light and infrared light change a tinge of color compared to an image captured by means of only visible light due to an influence including infrared light.
By the way, to use one device both for capturing images by means of visible light and capturing images by means of infrared light, this device may be configured to include a detachable mechanism of attaching and detaching an infrared cut filter to and from an imaging device. However, when the detaching mechanism is provided, the imaging device becomes larger, and manufacturing cost increases. For mobile terminals such as mobile telephones and smart phones which have cameras in particular, an increase in the size of devices is a problem.
Hence, a technique of correcting a color of an image captured by means of visible light and infrared light using a matrix operation without an infrared cut filter is proposed.    Patent Literature 1: Japanese Laid-open Patent Publication No. 2006-094112    Patent Literature 2: Japanese Laid-open Patent Publication No. 2006-101231    Patent Literature 3: Japanese Laid-open Patent Publication No. 2005-295419    Patent Literature 4: Japanese Laid-open Patent Publication No. 2003-110860    Non-Patent Literature 1: TOYODA Yoshitaka, et al. “Near Infrared Cameras to Capture Full Color Images—A Study of Color Reproduction Methods Without an Infrared Cut Filter for Digital Color Cameras-”, ITE Journal, Vol. 64, No 1, pp. 101-110, 2010
Hereinafter, a conventional example of color correction will be described. A value of each pixel of an image which indicates a color is determined based on a predetermined gradation such as 256 gradations in color spaces of R (Red), G (Green) and B (Blue). When an uncorrected color is X and a target color resulting from correction of the color X is Y, the uncorrected color X of N pixels and the target color Y are represented by a matrix of RGB 3 colors×N in following equation 1 and equation 2.
                    x        =                  [                                                                      Rx                  ⁢                                                                          ⁢                  1                                                                              Rx                  ⁢                                                                          ⁢                  2                                                                              Rx                  ⁢                                                                          ⁢                  3                                                                              Rx                  ⁢                                                                          ⁢                  4                                                            …                                                              R                  ⁢                                                                          ⁢                  xN                                                                                                      Gx                  ⁢                                                                          ⁢                  1                                                                              Bx                  ⁢                                                                          ⁢                  2                                                                              Bx                  ⁢                                                                          ⁢                  3                                                                              Gx                  ⁢                                                                          ⁢                  4                                                            …                                                              G                  ⁢                                                                          ⁢                  xN                                                                                                      Bx                  ⁢                                                                          ⁢                  1                                                                              Gx                  ⁢                                                                          ⁢                  2                                                                              Gx                  ⁢                                                                          ⁢                  3                                                                              Bx                  ⁢                                                                          ⁢                  4                                                            …                                                              B                  ⁢                                                                          ⁢                  xN                                                              ]                                    (        1        )                                y        =                  [                                                                      Ry                  ⁢                                                                          ⁢                  1                                                                              Ry                  ⁢                                                                          ⁢                  2                                                                              Ry                  ⁢                                                                          ⁢                  3                                                                              Ry                  ⁢                                                                          ⁢                  4                                                            …                                                              R                  ⁢                                                                          ⁢                  yN                                                                                                      Gy                  ⁢                                                                          ⁢                  1                                                                              By                  ⁢                                                                          ⁢                  2                                                                              By                  ⁢                                                                          ⁢                  3                                                                              Gy                  ⁢                                                                          ⁢                  4                                                            …                                                              G                  ⁢                                                                          ⁢                  yN                                                                                                      By                  ⁢                                                                          ⁢                  1                                                                              Gy                  ⁢                                                                          ⁢                  2                                                                              Gy                  ⁢                                                                          ⁢                  3                                                                              By                  ⁢                                                                          ⁢                  4                                                            …                                                              B                  ⁢                                                                          ⁢                  yN                                                              ]                                    (        2        )            
A correction coefficient B for correcting the uncorrected color X to the target color Y is set to the matrix of 3×3 as indicated in following equation 3.
                    B        =                  [                                                                      α                  ⁢                                                                          ⁢                  r                                                                              α                  ⁢                                                                          ⁢                  g                                                                              α                  ⁢                                                                          ⁢                  b                                                                                                      β                  ⁢                                                                          ⁢                  r                                                                              β                  ⁢                                                                          ⁢                  g                                                                              β                  ⁢                                                                          ⁢                  b                                                                                                      γ                  ⁢                                                                          ⁢                  r                                                                              γ                  ⁢                                                                          ⁢                  g                                                                              γ                  ⁢                                                                          ⁢                  b                                                              ]                                    (        3        )            
The target color Y is represented by an operation of the correction coefficient for the uncorrected color X as indicated in following equation 4.Y=B·X  (4)
The correction coefficient B can preferably convert the uncorrected color X close to the target color Y as much as possible. Hence, this correction coefficient B is determined by solving a minimization problem in following equation 5.min∥y−Bx∥2  (5)
The correction coefficient B is calculated as a pseudo inverse matrix as indicated in following equation 6. “xt” represents a transposed matrix of x.B=y·xt/x·xt  (6)
When a corrected color corrected from the uncorrected color X using the corrected coefficient B is Yout, Yout is represented as in following equation 7.Yout=B·x  (7)
Meanwhile, when an image is corrected using the correction coefficient B calculated according to equation 6, a phenomenon occurs that the corrected image includes significant noise.
Meanwhile, an example of the phenomenon that noise becomes significant will be described using an image obtained by capturing a color checker target made by X-Rite, Inc. FIG. 18 is a view illustrating an example of an image obtained by capturing a color checker target by an imaging device provided with an infrared cut filter. FIG. 19 is a view illustrating an example of an image obtained by capturing the color checker target by the imaging device without the infrared cut filter. As illustrated in FIGS. 18 and 19, a color checker target 200 has rectangular color sample areas 201 of 24 colors including a gray tone. FIG. 19 illustrates that a tinge of color changes compared to FIG. 18. This means that the infrared cut filter is not provided, and therefore infrared light is also incident on a light receiving unit of each color of RGB of an imaging element in addition to visible light and the tinge of color changes due to an influence of electrical charges resulting from the received infrared light.
By performing an operation indicated in equation 6 assuming each color in FIG. 18 is the target color Y and each color in FIG. 19 is the uncorrected color X, the correction coefficient B of converting the uncorrected color X into the target color Y is calculated. An example of the correction coefficient B calculated by performing the operation indicated in equation 6 is indicated in equation 8.
                    B        =                  (                                                    0.1140                                            3.2487                                                              -                  2.3859                                                                                                      -                  1.5110                                                            5.5897                                                              -                  2.7685                                                                                                      -                  1.4184                                                            4.2156                                                              -                  1.7509                                                              )                                    (        8        )            
Further, correction is performed by calculating a corrected color Yout per pixel by performing conversion indicated in equation 7 for the color X of each pixel of the image in FIG. 19 using the correction coefficient B. FIG. 20 is a view illustrating an example of an image obtained by correcting the image in FIG. 19 using the correction coefficient B. FIG. 20 illustrates there is more significant noise compared to FIG. 18.
Meanwhile, to clarify a change of an image resulting from correction, RGB values indicating colors are converted into brightness Y and color differences I and Q per corresponding pixel between the uncorrected image and the corrected image, and are compared. FIG. 21A is a graph illustrating an example where the brightness Y of corresponding pixels of the image in FIG. 18 and an image in FIG. 20 are compared. With an example in FIG. 21A, the brightness Y of each pixel along a broken line in FIG. 18 is indicated as a target value, and the brightness Y of each pixel along a broken line in FIG. 20 is indicated as a correction value. FIG. 21B is a graph illustrating an example where the color difference I between corresponding pixels of the image in FIG. 18 and the image in FIG. 20 is compared. With an example in FIG. 21B, the color difference I of each pixel along the broken line in FIG. 18 is indicated as a target value, and the color difference I of each pixel along the broken line in FIG. 20 is indicated as a correction value. FIG. 21C is a graph illustrating an example where the color difference Q between corresponding pixels of the image in FIG. 18 and the image in FIG. 20 is compared. With an example in FIG. 21C, the color difference Q of each pixel along the broken line in FIG. 18 is indicated as a target value, and the color difference Q of each pixel along the broken line in FIG. 20 is indicated as a correction value.
As illustrated in FIGS. 21B and 21C, the color differences I and Q change such that a correction value changes close to a target value. That is, the change in the tinge of color resulting from correction is a little. Meanwhile, as illustrated in FIG. 21A, the correction value of the brightness Y significantly changes with respect to the target value. A reason that the correction value of the brightness Y significantly changes with respect to the target value is that, when a mix amount of infrared light for the imaging element is large, each element of the correction coefficient B is much larger than 1 and, as a result, noise in the corrected image is emphasized.
In order to reduce noise in the brightness Y, a weight is applied to the correction coefficient B. For example, a minimization problem will be described that, as indicated in following equation 9, a term of a weight coefficient λ is added to the correction coefficient B using a Tikhonov regularization method, and a norm of a target color Y and a correction color obtained by correcting the uncorrected color X using a correction coefficient B′ is regularized.min(∥y−B′x∥2+λ2∥B′∥2  (9)
This correction coefficient B′ is calculated according to following equation 10.B′=y·xt/(x·xt+λ2·I)  (10)
Meanwhile, I represents a unit matrix.
For example, equation 11 indicates an example of the correction coefficient B′ calculated by performing an operation indicated in equation 10 when λ=30 is true.
                    B        =                  (                                                    0.6786                                            0.7495                                                              -                  0.9007                                                                                                      -                  0.0724                                                            1.2677                                                              -                  0.6486                                                                                                      -                  0.2512                                                            0.9837                                                              -                  0.2544                                                              )                                    (        11        )            
An image is corrected by converting the color X of each pixel of the image in FIG. 19 using the correction coefficient B′ calculated according to equation 8 and calculating the corrected color Yout per pixel. FIG. 22 is a view illustrating an example of an image obtained by correcting the image in FIG. 19 using the correction coefficient B′. FIG. 22 illustrates that noise decreases compared to FIG. 20 and the tinge of color significantly changes compared to FIG. 18.
Meanwhile, to clarify a change of an image resulting from correction, RGB values indicating colors are converted into the brightness Y and color differences I and Q per corresponding pixel between the uncorrected image and the corrected image, and are compared. FIG. 23A is a graph illustrating an example where the brightness Y of the image in FIG. 18 and an image in FIG. 22 are compared. With an example in FIG. 23A, the brightness Y of each pixel along the broken line in FIG. 18 is indicated as a target value, and the brightness Y of each pixel along the broken line in FIG. 22 is indicated as a correction value. FIG. 23B is a graph illustrating an example where the color difference between corresponding pixels of the image in FIG. 18 and the image in FIG. 22 is compared. With an example in FIG. 23B, the color difference I of each pixel along the broken line in FIG. 18 is indicated as a target value, and the color difference I of each pixel along the broken line in FIG. 22 is indicated as a correction value. FIG. 23C is a graph illustrating an example where the color difference Q between corresponding pixels of the image in FIG. 18 and the image in FIG. 22 is compared. With an example in FIG. 23C, the color difference Q of each pixel along the broken line in FIG. 18 is indicated as a target value, and the color difference Q of each pixel along the broken line in FIG. 22 is indicated as a correction value.
As illustrated in FIG. 23A, a correction value of the brightness Y changes close to the target value and in a small range. That is, noise resulting from correction is a little. Meanwhile, as illustrated in FIGS. 23B and 23C, correction values of the color differences I and Q change from the target values. That is, the tinge of color changes due to correction.
Thus, while the corrected brightness Y includes significant noise when a weight is not applied the color differences I and Q are close to the target values. Meanwhile, when a weight is applied, while the corrected brightness Y includes little noise, the color differences I and Q are different. As described above, the noise and the tinge of color have a trade-off relationship.